Robustness Measurement: an Approach to Assessing Simulation Program Reliability

We describe a new approach to measuring the quality of simulation software by measuring its robustness. Robustness is the ability to give similar or identical numerical results even when mutations | random bugs | are injected into the code. Robustness is important because large programs, such as large simulation programs, always or nearly always have unintended bugs. Thus it is desirable for a program to behave similarly in the presence of bugs to the way it would behave without bugs. We estimate how well the results of an existing simulation program p 0 may be expected to match the results of its bug free counterpart p by intentionally adding bugs to p 0 , producing a set of programs P 1 containing programs p 1;j that each di ers from p 0 in containing a single added bug. Each program p 1;j in P 1 is further mutated to provide a 2-bug program p 2;j in 2-bug program set P 2 . Likewise, a set P 3 with three bugs in each member is created, etc. Then, we measure how closely the the simulation results predicted by programs in P N match those predicted by program p 0 : These robustness results then constitute knowledge about the relative dependability of p 0 : Robust simulation programs give us more con dence than non-robust ones. Readers of this paper are requested to contact us regarding robustness testing of their simulations.